{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/4-Evaluating%20LLMs/4_3_evaluating_rag_giskard.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f3cvnSVemTSU"
   },
   "source": [
    "<div>\n",
    "    <h1>Large Language Models Projects</a></h1>\n",
    "    <h3>Apply and Implement Strategies for Large Language Models</h3>\n",
    "    <h2>4.3-Evaluating LLMs with other LLMs.\n",
    "  </h2>\n",
    "    <h3>Evaluating a RAG solution with Giskard</h3>\n",
    "</div>\n",
    "\n",
    "by [Pere Martra](https://www.linkedin.com/in/pere-martra)\n",
    "\n",
    "____________\n",
    "Models: OpenAI.\n",
    "\n",
    "Colab Environment: CPU-High RAM\n",
    "\n",
    "Keys:\n",
    "* LangChain.\n",
    "* RAG.\n",
    "* Giskard.\n",
    "___________\n",
    "\n",
    "This is the unofficial repository for the book:\n",
    "        <a href=\"https://amzn.to/4eanT1g\"> <b>Large Language Models:</b> Apply and Implement Strategies for Large Language Models</a> (<a href=\"https://link.springer.com/book/10.1007/979-8-8688-0515-8\">Apress</a>).\n",
    "        The book is based on the content of this repository, but the notebooks are being updated, and I am incorporating new examples and chapters.\n",
    "        If you are looking for the official repository for the book, with the original notebooks, you should visit the\n",
    "        <a href=\"https://github.com/Apress/Large-Language-Models-Projects\">Apress repository</a>, where you can find all the notebooks in their original format as they appear in the book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DUORXVgTmNYj"
   },
   "source": [
    "In the final part of this notebook, you will find the necessary code to evaluate the suitability of the responses provided by the Agent using Giskard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KNW-LniD2I2o"
   },
   "source": [
    "#Installing libraries & Loading Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pva9ehKXUpU2",
    "outputId": "699cf75c-7e27-4676-8305-d0ee21621709"
   },
   "outputs": [],
   "source": [
    "!pip install -q langchain==0.3.2\n",
    "!pip install -q langchain-openai==0.2.1\n",
    "!pip install -q langchainhub==0.1.21\n",
    "!pip install -q datasets==3.0.1\n",
    "!pip install -q chromadb==0.5.11\n",
    "!pip install -q openai==1.51.0\n",
    "!pip install -q langchain-community==0.3.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KEwdHiGFkK_S"
   },
   "source": [
    "We will download the dataset from the Hugging Face datasets library. It's a dataset with information about diseases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 113,
     "referenced_widgets": [
      "4b90b500fe274ce3804d69633e3ce045",
      "f13720f8c0054b37ad63800962c01bd8",
      "a4689ab634be40769ff8c18d3076fbfd",
      "6850ddb41cdb47b3a42117ab825ddf55",
      "def97bfc99c843fcaefeec857804e9a7",
      "2341ca0e2acc46aab7755fab0442d7ec",
      "b6229a3e1b404dc1ae23a13d599f2b1a",
      "ce0ebb7981da496982ecb22864c9a8f8",
      "5174e8de97ad4bf8a6634ff749b61dff",
      "094b862e70a64a8bbe80625e9aacdd77",
      "834b7b72458341949259d352a681a2d5",
      "c3d158dc007941e0bacf1ada9b4d7959",
      "79b2ac0b24bf4dad8babf7b645f8bb84",
      "c933fc46c07440e288dccbd1c349f0a3",
      "1db2aec4c4fe469387537c1ad9eb4137",
      "9a7c82cff6374dadb1185d8a5653fe22",
      "63fac17cc9f444278a3ed1891fb5367f",
      "fc51b487afad4be48fe37f9d5cdf8e09",
      "44ccb08744c54d0d8b1a369b00110241",
      "8aa404c0ddda4b47955ca5b198d63604",
      "0a668f16bf584d65a36a1a1f8b32a045",
      "81591a2f532547e5ac1db6e900e86c3e",
      "6455d0b54a6d4c37b8a4c7e35ad170df",
      "a4fb5033245f46d4b9376e38956bd635",
      "4bb13b7ac5c64ae2a12f03e569320542",
      "be209ee684de4813a63e15c5a562c0f1",
      "9af4090ffa2f4e178f7abda763e091e0",
      "ab796022b81b4ee3b4f105f88f722453",
      "8051d6bec5864a3f82a3949576d51669",
      "16e3ee920a5d4342b93e4e4b8868c700",
      "58bc373738074a51bbe73777881d3179",
      "0b13b00d71ff438da830af5d1b9bc7a0",
      "4fc9af1de5904111a55f93fe0493d003"
     ]
    },
    "id": "laSDMjqQXuj-",
    "outputId": "368d5afd-3597-4057-a051-4244a8ab1c7a"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "data = load_dataset(\"keivalya/MedQuad-MedicalQnADataset\", split='train')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "JnWZTcJiXzor",
    "outputId": "3517ef47-d652-4c82-9a3a-715708657f7c"
   },
   "outputs": [],
   "source": [
    "data = data.to_pandas()\n",
    "data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hf7RQa6B5xpx"
   },
   "outputs": [],
   "source": [
    "data = data[0:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4FhUslovtiqn"
   },
   "source": [
    "As you can see, the medical information in the dataset is well-organized, and to someone like me, who is not an expert in the field, it appears to be quite valuable. This information could be a useful addition to any general medicine book to support primary care doctors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2sLkrHF6lWhM"
   },
   "source": [
    "Load the langchain libraries to load the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cCBAlIb596wZ"
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import DataFrameLoader\n",
    "from langchain.vectorstores import Chroma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nGCkVX6xldOR"
   },
   "source": [
    "The Document is in the Answer column, and the others columns are Metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JZX8SaTe99Uf"
   },
   "outputs": [],
   "source": [
    "df_loader = DataFrameLoader(data, page_content_column=\"Answer\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 481
    },
    "id": "HDWnTqRY-IDr",
    "outputId": "65a76c4b-e424-4406-d721-5444802f0e02"
   },
   "outputs": [],
   "source": [
    "df_document = df_loader.load()\n",
    "display(df_document[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zPuTbSsXl3uF"
   },
   "source": [
    "We can chunk the documents. The size to which we want to split the document is a design decision. The larger it is, the larger the prompt will be, and the slower the Model's response process.\n",
    "\n",
    "We also need to consider the maximum prompt size and ensure that the document does not exceed it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8wogWV1_-lxh"
   },
   "outputs": [],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kg7SRkqO-f9x"
   },
   "outputs": [],
   "source": [
    "text_splitter = CharacterTextSplitter(chunk_size=1250, chunk_overlap=100)\n",
    "texts = text_splitter.split_documents(df_document)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y0dn9EfrKoeW"
   },
   "source": [
    "These warnings we see are because it can't perform the partition of the required size. This is because it waits for a page break to divide the text and does so when possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XaLWV_TzASSB",
    "outputId": "8d41dee1-012f-4e8c-dc1f-40193f08fc77"
   },
   "outputs": [],
   "source": [
    "first_doc = texts[1]\n",
    "print(first_doc.page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B2_Pt7N6Zg2X"
   },
   "source": [
    "### Initialize the Embedding Model and Vector DB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n8ROb8oMnRLD"
   },
   "source": [
    "We load the text-embedding-ada-002 model from OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F_Dn06xGwjKP",
    "outputId": "0464a93d-63ad-48eb-8d46-32850791bb61"
   },
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "import os\n",
    "if not 'OPENAI_API_KEY' in os.environ:\n",
    "  os.environ[\"OPENAI_API_KEY\"] = getpass(\"OpenAI API Key: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U57x2_87YSpb"
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "model_name = 'text-embedding-ada-002'\n",
    "#model_name = 'text-embedding-3-small'\n",
    "\n",
    "embed = OpenAIEmbeddings(\n",
    "    model=model_name,\n",
    "    #openai_api_key=OPENAI_API_KEY\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cgTCwF7UMyNW"
   },
   "source": [
    "The execution of this cell may take 3 to 5 minutes. If you want it to be faster, you can reduce the number of records in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SEhQMQ8eCMj8"
   },
   "outputs": [],
   "source": [
    "directory_cdb = '/content/drive/MyDrive/chromadb'\n",
    "chroma_db = Chroma.from_documents(\n",
    "    df_document, embed, persist_directory=directory_cdb\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KhjV-T8GoarF"
   },
   "source": [
    "We are going to create three objects.\n",
    "\n",
    "* The language model, which can be any of those from OpenAI, the most common being gpt-3.5.\n",
    "* The memory, responsible for keeping the prompt with all the necessary history.\n",
    "* The retrieval, used to obtain information stored in ChromaDB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zMRs9Klic5-Y",
    "outputId": "b3988759-8c38-4458-b3ee-bbad9593d871"
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain_openai import OpenAI\n",
    "from langchain.chains.conversation.memory import ConversationBufferWindowMemory\n",
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "llm=OpenAI(#openai_api_key=OPENAI_API_KEY,\n",
    "           temperature=0.0)\n",
    "\n",
    "conversational_memory = ConversationBufferWindowMemory(\n",
    "    memory_key='chat_history',\n",
    "    k=4, #Number of messages stored in memory\n",
    "    return_messages=True #Must return the messages in the response.\n",
    ")\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm,\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=chroma_db.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ergrieE4o8qu"
   },
   "source": [
    "We can try the isolated Retrieval to see if the information it returns is relevant.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LaYSq0V-dxHw",
    "outputId": "99e89396-1ba4-4c46-bfdc-c9e851d4aebd"
   },
   "outputs": [],
   "source": [
    "qa.invoke(\"What is the main symptom of LCM?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pf9MXPeipEBO"
   },
   "source": [
    "Perfect! The information returned is exactly what we desired.\n",
    "\n",
    "## Creating the Agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FwCYrS4duqBW"
   },
   "outputs": [],
   "source": [
    "from langchain.agents import Tool, AgentExecutor\n",
    "DESCRIPTION = 'use this tool when answering medical knowledge queries to get more information about the topic'\n",
    "#DESCRIPTION = 'Use this tool if any part of the QUERY or CHAT HISTORY history discusses concepts related to medicine or diseases to get more information about the topic'\n",
    "\n",
    "#Defining the list of tool objects to be used by LangChain.\n",
    "tools = [\n",
    "    Tool(\n",
    "        name='Medical KB',\n",
    "        func=qa.run,\n",
    "        description=(\n",
    "            DESCRIPTION\n",
    "        )\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JaKTzPUEvOoy",
    "outputId": "787a391e-8aa6-49a4-c59c-a2e0ca341790"
   },
   "outputs": [],
   "source": [
    "from langchain.agents import initialize_agent, create_react_agent\n",
    "from langchain import hub\n",
    "\n",
    "prompt = hub.pull(\"hwchase17/react-chat\")\n",
    "agent = create_react_agent(\n",
    "    #agent='chat-conversational-react-description',\n",
    "    tools=tools,\n",
    "    llm=llm,\n",
    "    prompt=prompt,\n",
    "    #verbose=True,\n",
    "    #max_iterations=3,\n",
    "    #early_stopping_method='generate'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5sjTL1EU1vNW"
   },
   "outputs": [],
   "source": [
    "# Create an agent executor by passing in the agent and tools\n",
    "agent_executor = AgentExecutor(agent=agent,\n",
    "                               tools=tools,\n",
    "                               verbose=True,\n",
    "                               memory=conversational_memory,\n",
    "                               max_iterations=30,\n",
    "                               max_execution_time=600,\n",
    "                               #early_stopping_method='generate',\n",
    "                               handle_parsing_errors=True\n",
    "                               )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IlxUBWKcvzeP"
   },
   "source": [
    "### Using the Conversational Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZZapCP4Pv2kz"
   },
   "source": [
    "To make queries we simply call the `agent` directly.\n",
    "\n",
    "First i will try a order not related to the Medical field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "85vipqC02deV",
    "outputId": "c02aba7a-6fca-41df-826b-d2a55a6a6ebf"
   },
   "outputs": [],
   "source": [
    "agent_executor.invoke({\"input\": \"Give me the area of square of 2x2\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8suStMR7G11e"
   },
   "source": [
    "Perfect, the model has responded without accessing the configured knowledge database.\n",
    "\n",
    "Now I will try with a question that is also not related to health."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9YEsxEuCVgMv",
    "outputId": "95ab9e3f-fc2d-48e2-a7c4-b92aaebcb31e"
   },
   "outputs": [],
   "source": [
    "agent_executor.invoke({\"input\": \"Do you know who is Clark Kent?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lyd0XJ3CHjVb"
   },
   "source": [
    "It has not accessed either, as the model has been able to identify that it is not a question related to the database that LangChain provides.\n",
    "\n",
    "Now it's time to try with a question related to Medicine. Let's see if the model can understand that it should first look for information in the vector database at its disposal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wtwgfuy158LV"
   },
   "outputs": [],
   "source": [
    " agent_executor.memory.clear()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RJoAhy76vzAB",
    "outputId": "d4148000-d8da-47ff-d13d-e3f142b5c2f3"
   },
   "outputs": [],
   "source": [
    "agent_executor.invoke({\"input\": \"\"\"I have a patient that can have Botulism,\n",
    "how can I confirm the diagnose?\"\"\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8lsaoF8nJNnR"
   },
   "source": [
    "Perfect, the most important thing for us is that it has been able to identify that it should go to the medical database to search for information about the symptoms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mQeicHTj2pmY",
    "outputId": "5b5de623-2e41-4c29-dc11-a676f2573489"
   },
   "outputs": [],
   "source": [
    "agent_executor.invoke({\"input\": \"Is this an important illness?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "quf3qyI_aVz3"
   },
   "source": [
    "And the memory works perfectly. We can maintain a conversation, taking into account that the model knows the previous questions and answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FQwHcAMi3e85"
   },
   "source": [
    "## Evaluating the solution with Giskard\n",
    "\n",
    "Install and Load the libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6Em-qkfK637x",
    "outputId": "2f8ad0d6-2238-4206-a31d-f96019ee5e35"
   },
   "outputs": [],
   "source": [
    "!pip install -q giskard[llm]==2.15.1\n",
    "from giskard.rag import KnowledgeBase, generate_testset, evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qXN4hGqDoMg1"
   },
   "source": [
    "Is necesary to create a Dataframe with just the column containing the information used to create the RAG system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "yeIMrHlhMJDF",
    "outputId": "a78935d6-2971-411e-d6e3-b1843edc0c6c"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df_giskard = pd.DataFrame([d.page_content for d in df_document], columns=[\"text\"])\n",
    "df_giskard.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VK3hgB-MoWLj"
   },
   "source": [
    "Using the information from the dataset, we ask Giskard to create a Knowledge Base, which is nothing more than a set of questions along with their respective answers. Both the questions and answers are generated by OpenAI's most advanced model, which is why it requires our OpenAI key to be provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iPaw4w9c_Gbo"
   },
   "outputs": [],
   "source": [
    "kb_giskard = KnowledgeBase(df_giskard)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xewqzBgzwrOv",
    "outputId": "3ca334e8-fe78-44fb-d10a-89ee1f531f8e"
   },
   "outputs": [],
   "source": [
    "kb_giskard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85,
     "referenced_widgets": [
      "402bc040234745f8ae9a557728eb8354",
      "c1409fb9cd114ea1bcc5dd77ddc504e7",
      "1e3b95759e8a46e693599b340a0835f9",
      "4038dab690dc4e17a0123cd8bc07edcd",
      "04cdfa01853447449e6e58ae355e0687",
      "332819930be54450b2c4c66aa37fb3cd",
      "4e03497cf82e44b7bc1135d73e91d4f0",
      "82b885247c6f4cb2a826b6abc6d5175e",
      "d94c38c92b444e92b6e8a08c4c652e64",
      "ceb438b0126847ff9009604005732812",
      "372c3c4bcbf643ce9141a987125d8206"
     ]
    },
    "id": "VcSnDxElKQmw",
    "outputId": "8b44734a-0365-4d5b-84b5-cf1daf1afa14"
   },
   "outputs": [],
   "source": [
    "# The more questions you generate, the more you will be charged.\n",
    "test_questions = generate_testset(\n",
    "    kb_giskard,\n",
    "    num_questions=10,\n",
    "    agent_description=\"Medical assistant for diagnosis and treatment support.\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rYH3J0MXTJl1"
   },
   "outputs": [],
   "source": [
    "df_test_questions = test_questions.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 411
    },
    "id": "rvHhO_oDThft",
    "outputId": "b4778f6b-69f8-43ac-e65d-0d60e1ffcf56"
   },
   "outputs": [],
   "source": [
    "df_test_questions.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "i1nOfJ7pSEKg",
    "outputId": "457b6272-615a-4a3c-d06b-58dc30095485"
   },
   "outputs": [],
   "source": [
    "df_test_questions['question'].iloc[1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "_nzEw9rNTuW2",
    "outputId": "de0f9700-358f-4aed-f5fd-c2274738907c"
   },
   "outputs": [],
   "source": [
    "df_test_questions['reference_answer'].iloc[1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "id": "RcT53F6ZTvcs",
    "outputId": "ae54b62e-518d-48cd-a53a-e6475efe354b"
   },
   "outputs": [],
   "source": [
    "df_test_questions['reference_context'].iloc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "21tFRKfXC2wF"
   },
   "source": [
    "A function is created that will be called from Giskard's evaluate function. This function receives the question and returns the agent's response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "InRbIK2_V2HU"
   },
   "outputs": [],
   "source": [
    "def use_agent(question, history=None):\n",
    "  answer=agent_executor.invoke({\"input\": question})\n",
    "  return answer['output']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DRuhL8XMXK-I"
   },
   "outputs": [],
   "source": [
    "agent_executor.memory.clear()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "0f0e0eb678fb4561ae67c8a56d0024f8",
      "f94992c7afee4b1591db5164cf107880",
      "9e00a1effe084ab9806fb7aab4775984",
      "cd7396bfa8df48b2864a109ce7c66a2c",
      "8f5ef7b50a3c43609c7c1a168e6ac9b6",
      "7fce24a65eb74ec8bec35caefedc3107",
      "cad8096da6874fe0898f1345c1e7bbf4",
      "3b083846d432457782171a810b460771",
      "ae34ed2aae744624b048ec526413dbdf",
      "6b8a6330b0b74c649176611711b99231",
      "f3ad037f417c46b5a3e52e86486c379a",
      "ddafa25f88ca47fb81172779c34ca1aa",
      "2885e20894324970a12f52d3cd7f15b1",
      "98ffcb3dc9bf4cf6bdb8a00e57f73a55",
      "6f234f8281c4445792dc4ea0ce7a7b9e",
      "840a177b18254751b07f564176429e36",
      "9806352a846a4fec9c6fecd2aa3fc1ad",
      "c024e5f9f06342169b7f7c4a5327d7ff",
      "a66f8156f0d6406cab082b74b1b74aa2",
      "b023814d0cf04b589926d9f64cd72ad3",
      "2ec7a5d413864daab9f68dbdd30b04b2",
      "33297c2c5e1d4ad0a3fe95d7bff4cb46"
     ]
    },
    "id": "D3fm-_7RU4y9",
    "outputId": "acadcb6c-cd92-4367-d4d5-c0dcd0c5a6ed"
   },
   "outputs": [],
   "source": [
    "report = evaluate(use_agent, testset=test_questions, knowledge_base=kb_giskard)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "WNLuWCB2YJwO",
    "outputId": "2664b76d-67d7-423c-e980-cb15e9f1fa85"
   },
   "outputs": [],
   "source": [
    "# Summary with the results.\n",
    "report.correctness_by_question_type()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "id": "-ZR-AoQyXBCP",
    "outputId": "e3b4fb9f-33ad-4887-cce3-c7ba880058dd"
   },
   "outputs": [],
   "source": [
    "report.correctness_by_topic()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "id": "XgHVO7jBYl-3",
    "outputId": "ae885f6d-def6-4042-ffc6-95bc80cd7068"
   },
   "outputs": [],
   "source": [
    "report.component_scores()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 320
    },
    "id": "EsBU-rL4d1lM",
    "outputId": "b38acc04-4a4b-4a3d-d64c-24569ca63b43"
   },
   "outputs": [],
   "source": [
    "# Obtaining the incorrect answers\n",
    "failures = report.get_failures()[:2]\n",
    "failures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "id": "VBFOYCMTeOUd",
    "outputId": "20e4f2df-2fbf-485b-b574-4e1a9b8daa08"
   },
   "outputs": [],
   "source": [
    "# Giskard explains the reasons why it considers the answers to be incorrect.\n",
    "failures['correctness_reason'].iloc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gdyhyd6nJnwN"
   },
   "source": [
    "\n",
    "# Conclusions.\n",
    "The experiment has been a small success. The Vectorial database has been configured and filled with information from the dataset. A LangChain agent has been created, and it has been able to retrieve information from the database only when necessary. Don't forget that our ChatBot has memory.\n",
    "\n",
    "All of this in just a few lines of code!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ykg5TYA033yR"
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "gpuType": "V100",
   "include_colab_link": true,
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.9.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "04cdfa01853447449e6e58ae355e0687": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "094b862e70a64a8bbe80625e9aacdd77": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0a668f16bf584d65a36a1a1f8b32a045": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0b13b00d71ff438da830af5d1b9bc7a0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0f0e0eb678fb4561ae67c8a56d0024f8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f94992c7afee4b1591db5164cf107880",
       "IPY_MODEL_9e00a1effe084ab9806fb7aab4775984",
       "IPY_MODEL_cd7396bfa8df48b2864a109ce7c66a2c"
      ],
      "layout": "IPY_MODEL_8f5ef7b50a3c43609c7c1a168e6ac9b6"
     }
    },
    "16e3ee920a5d4342b93e4e4b8868c700": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1db2aec4c4fe469387537c1ad9eb4137": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0a668f16bf584d65a36a1a1f8b32a045",
      "placeholder": "​",
      "style": "IPY_MODEL_81591a2f532547e5ac1db6e900e86c3e",
      "value": " 22.5M/22.5M [00:02&lt;00:00, 8.77MB/s]"
     }
    },
    "1e3b95759e8a46e693599b340a0835f9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_82b885247c6f4cb2a826b6abc6d5175e",
      "max": 10,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_d94c38c92b444e92b6e8a08c4c652e64",
      "value": 10
     }
    },
    "2341ca0e2acc46aab7755fab0442d7ec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2885e20894324970a12f52d3cd7f15b1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9806352a846a4fec9c6fecd2aa3fc1ad",
      "placeholder": "​",
      "style": "IPY_MODEL_c024e5f9f06342169b7f7c4a5327d7ff",
      "value": "CorrectnessMetric evaluation: 100%"
     }
    },
    "2ec7a5d413864daab9f68dbdd30b04b2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "332819930be54450b2c4c66aa37fb3cd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "33297c2c5e1d4ad0a3fe95d7bff4cb46": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "372c3c4bcbf643ce9141a987125d8206": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3b083846d432457782171a810b460771": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "402bc040234745f8ae9a557728eb8354": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c1409fb9cd114ea1bcc5dd77ddc504e7",
       "IPY_MODEL_1e3b95759e8a46e693599b340a0835f9",
       "IPY_MODEL_4038dab690dc4e17a0123cd8bc07edcd"
      ],
      "layout": "IPY_MODEL_04cdfa01853447449e6e58ae355e0687"
     }
    },
    "4038dab690dc4e17a0123cd8bc07edcd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ceb438b0126847ff9009604005732812",
      "placeholder": "​",
      "style": "IPY_MODEL_372c3c4bcbf643ce9141a987125d8206",
      "value": " 10/10 [01:23&lt;00:00,  9.44s/it]"
     }
    },
    "44ccb08744c54d0d8b1a369b00110241": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b90b500fe274ce3804d69633e3ce045": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f13720f8c0054b37ad63800962c01bd8",
       "IPY_MODEL_a4689ab634be40769ff8c18d3076fbfd",
       "IPY_MODEL_6850ddb41cdb47b3a42117ab825ddf55"
      ],
      "layout": "IPY_MODEL_def97bfc99c843fcaefeec857804e9a7"
     }
    },
    "4bb13b7ac5c64ae2a12f03e569320542": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_16e3ee920a5d4342b93e4e4b8868c700",
      "max": 16407,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_58bc373738074a51bbe73777881d3179",
      "value": 16407
     }
    },
    "4e03497cf82e44b7bc1135d73e91d4f0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4fc9af1de5904111a55f93fe0493d003": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5174e8de97ad4bf8a6634ff749b61dff": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "58bc373738074a51bbe73777881d3179": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "63fac17cc9f444278a3ed1891fb5367f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6455d0b54a6d4c37b8a4c7e35ad170df": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a4fb5033245f46d4b9376e38956bd635",
       "IPY_MODEL_4bb13b7ac5c64ae2a12f03e569320542",
       "IPY_MODEL_be209ee684de4813a63e15c5a562c0f1"
      ],
      "layout": "IPY_MODEL_9af4090ffa2f4e178f7abda763e091e0"
     }
    },
    "6850ddb41cdb47b3a42117ab825ddf55": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_094b862e70a64a8bbe80625e9aacdd77",
      "placeholder": "​",
      "style": "IPY_MODEL_834b7b72458341949259d352a681a2d5",
      "value": " 233/233 [00:00&lt;00:00, 17.1kB/s]"
     }
    },
    "6b8a6330b0b74c649176611711b99231": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6f234f8281c4445792dc4ea0ce7a7b9e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2ec7a5d413864daab9f68dbdd30b04b2",
      "placeholder": "​",
      "style": "IPY_MODEL_33297c2c5e1d4ad0a3fe95d7bff4cb46",
      "value": " 10/10 [00:23&lt;00:00,  2.71s/it]"
     }
    },
    "79b2ac0b24bf4dad8babf7b645f8bb84": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_63fac17cc9f444278a3ed1891fb5367f",
      "placeholder": "​",
      "style": "IPY_MODEL_fc51b487afad4be48fe37f9d5cdf8e09",
      "value": "medDataset_processed.csv: 100%"
     }
    },
    "7fce24a65eb74ec8bec35caefedc3107": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8051d6bec5864a3f82a3949576d51669": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "81591a2f532547e5ac1db6e900e86c3e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "82b885247c6f4cb2a826b6abc6d5175e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "834b7b72458341949259d352a681a2d5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "840a177b18254751b07f564176429e36": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8aa404c0ddda4b47955ca5b198d63604": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8f5ef7b50a3c43609c7c1a168e6ac9b6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9806352a846a4fec9c6fecd2aa3fc1ad": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "98ffcb3dc9bf4cf6bdb8a00e57f73a55": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a66f8156f0d6406cab082b74b1b74aa2",
      "max": 10,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b023814d0cf04b589926d9f64cd72ad3",
      "value": 10
     }
    },
    "9a7c82cff6374dadb1185d8a5653fe22": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9af4090ffa2f4e178f7abda763e091e0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9e00a1effe084ab9806fb7aab4775984": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3b083846d432457782171a810b460771",
      "max": 10,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ae34ed2aae744624b048ec526413dbdf",
      "value": 10
     }
    },
    "a4689ab634be40769ff8c18d3076fbfd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ce0ebb7981da496982ecb22864c9a8f8",
      "max": 233,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5174e8de97ad4bf8a6634ff749b61dff",
      "value": 233
     }
    },
    "a4fb5033245f46d4b9376e38956bd635": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ab796022b81b4ee3b4f105f88f722453",
      "placeholder": "​",
      "style": "IPY_MODEL_8051d6bec5864a3f82a3949576d51669",
      "value": "Generating train split: 100%"
     }
    },
    "a66f8156f0d6406cab082b74b1b74aa2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ab796022b81b4ee3b4f105f88f722453": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ae34ed2aae744624b048ec526413dbdf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b023814d0cf04b589926d9f64cd72ad3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b6229a3e1b404dc1ae23a13d599f2b1a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "be209ee684de4813a63e15c5a562c0f1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0b13b00d71ff438da830af5d1b9bc7a0",
      "placeholder": "​",
      "style": "IPY_MODEL_4fc9af1de5904111a55f93fe0493d003",
      "value": " 16407/16407 [00:00&lt;00:00, 47610.71 examples/s]"
     }
    },
    "c024e5f9f06342169b7f7c4a5327d7ff": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c1409fb9cd114ea1bcc5dd77ddc504e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_332819930be54450b2c4c66aa37fb3cd",
      "placeholder": "​",
      "style": "IPY_MODEL_4e03497cf82e44b7bc1135d73e91d4f0",
      "value": "Generating questions: 100%"
     }
    },
    "c3d158dc007941e0bacf1ada9b4d7959": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_79b2ac0b24bf4dad8babf7b645f8bb84",
       "IPY_MODEL_c933fc46c07440e288dccbd1c349f0a3",
       "IPY_MODEL_1db2aec4c4fe469387537c1ad9eb4137"
      ],
      "layout": "IPY_MODEL_9a7c82cff6374dadb1185d8a5653fe22"
     }
    },
    "c933fc46c07440e288dccbd1c349f0a3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_44ccb08744c54d0d8b1a369b00110241",
      "max": 22466890,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_8aa404c0ddda4b47955ca5b198d63604",
      "value": 22466890
     }
    },
    "cad8096da6874fe0898f1345c1e7bbf4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cd7396bfa8df48b2864a109ce7c66a2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6b8a6330b0b74c649176611711b99231",
      "placeholder": "​",
      "style": "IPY_MODEL_f3ad037f417c46b5a3e52e86486c379a",
      "value": " 10/10 [00:35&lt;00:00,  3.66s/it]"
     }
    },
    "ce0ebb7981da496982ecb22864c9a8f8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ceb438b0126847ff9009604005732812": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d94c38c92b444e92b6e8a08c4c652e64": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "ddafa25f88ca47fb81172779c34ca1aa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2885e20894324970a12f52d3cd7f15b1",
       "IPY_MODEL_98ffcb3dc9bf4cf6bdb8a00e57f73a55",
       "IPY_MODEL_6f234f8281c4445792dc4ea0ce7a7b9e"
      ],
      "layout": "IPY_MODEL_840a177b18254751b07f564176429e36"
     }
    },
    "def97bfc99c843fcaefeec857804e9a7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f13720f8c0054b37ad63800962c01bd8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2341ca0e2acc46aab7755fab0442d7ec",
      "placeholder": "​",
      "style": "IPY_MODEL_b6229a3e1b404dc1ae23a13d599f2b1a",
      "value": "README.md: 100%"
     }
    },
    "f3ad037f417c46b5a3e52e86486c379a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f94992c7afee4b1591db5164cf107880": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7fce24a65eb74ec8bec35caefedc3107",
      "placeholder": "​",
      "style": "IPY_MODEL_cad8096da6874fe0898f1345c1e7bbf4",
      "value": "Asking questions to the agent: 100%"
     }
    },
    "fc51b487afad4be48fe37f9d5cdf8e09": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
